refactor: remove unused cast compatibility flag - #6135

Thanks for picking this up. The cleanup reads nicely, and the reserved lines match how we retire proto fields elsewhere. native/core doesn't compile with this change though. schema_adapter.rs:1353 still passes self.parquet_options.allow_incompat as a third argument to SparkCastOptions::new, and cargo check --workspace --all-targets fails with error[E0061]: this function takes 2 arguments but 3 arguments were supplied. Could you drop that argument? Running cargo clippy --all-targets --workspace -- -D warnings from native/ is what CI runs, so it will catch anything else across the crates.

Once that argument is gone, SparkParquetOptions::allow_incompat in parquet_support.rs has no reader left. The production constructors in parquet_exec.rs and iceberg_scan.rs both pass false, and the field still has the "Allow casts that are supported but not guaranteed to be 100% compatible" doc comment. That leaves readers with the same wrong impression #5077 is trying to remove. Would you be up for removing it from SparkParquetOptions and its two constructors in this PR too? Most of the call sites are tests.

Summary

• Prior state and problem: allowIncompatible was enforced during JVM planning but also serialized and carried through unused native options.
• Design approach: Remove that dead state from cast and Parquet options without introducing another abstraction.
• Correctness / compatibility analysis: The production cast and Parquet conversion kernels are unchanged. JVM compatibility gating, eval modes, timezones, and Spark-version handling remain intact. Checked relevant Spark sources for 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0.
• Key design decisions: Reserving protobuf field 5 and allow_incompat preserves the remaining wire contract. The change adds no execution work or allocations.
• Implementation sketch: Update serializers, constructors, planner and scan callers, tests, benchmarks, and audit documentation. Remove unused CometEvalMode helpers. The previously reported constructor-arity and unused-import concerns are fixed.
• Behavioral changes worth calling out: No cast-result or fallback-policy changes identified. No introduced P1/P2 issues found within this review.
• Suggested improvements: None at P1/P2 priority.

Reviewed all 35 changed files at bd3ac92f8ad1d12248bac645e1aee81566cc7559 against 7ee863ed5096049844cd05c635bc254cbaded804, including both cleanup commits. Confirmed non-draft status and read all supplied discussion. Routed skills: review-comet-pr and review-comet-expression-pr.

Exact-head CI: Only the labeling job passed. Comet CI and CodeQL report action_required, providing no build or test verdict.

Validation: cargo check --locked --offline --workspace --all-targets passed. Protobuf checks passed 36 old-to-new and 18 new-to-old cases. The focused native conversion-test attempt reached its 180-second limit while compiling dependencies, before tests ran. JVM, Spark SQL, and Iceberg suites were not run. Runtime validation therefore remains outstanding.

Thanks for the updates. The constructor arity errors, the unused import and the leftover SparkParquetOptions::allow_incompat are all addressed, and I no longer see a three-argument call in any test module. I have one question about scope on the new commit cefe845a1.

Thanks for the updates. The earlier items are all addressed, including the DPP change coming out of this PR.

One more break came in from main that you couldn't have seen. #6648 added native/spark-expr/benches/cast_nested.rs on 10-06, after your last merge, and it still passes the removed third argument to SparkCastOptions::new. Merged with this PR, cargo clippy --all-targets, which CI runs, fails with E0061 on that line. I merged the latest main into this branch and dropped the argument in a separate commit (66e7c77) so the fix is easy to see. Nothing else changed.

I ran the Rust unit tests, the Spark 3.5 scalafix check, and the cast, temporal and Parquet read suites on a local merge with main. On the pushed head I then ran clippy with -D warnings, cargo check --benches and cargo fmt --check. All of them pass. I also checked branch-1.1 and main for a native read of allow_incompat and found none, so the flag was dead on the native side and this is a pure removal.